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I.  Introduction 

Over  the  next  decade  the  Federal  Aviation  Administration 
(FAA)  is  planning  a  wide  ranging  transformation  of  the 
national  air  transportation  system,  known  as  NextGen.  With 
NextGen  the  FAA  will  move  away  from  single  sensor  ground 
based  surveillance  systems  towards  a  satellite  based  surveil¬ 
lance  system  known  as  Automatic  Dependent  Surveillance 
Broadcast  (ADS-B)  [1],  Currently  the  FAA  Order  7110.65 
detailing  the  surveillance  requirements  to  support  separation 
services  assumes  that  surveillance  is  provided  by  single  sensor 
radar  technology  [2].  To  extend  separation  services  to  other 
surveillance  systems,  their  performance  must  be  proven  equiv¬ 
alent  to  or  better  than  the  existing  radar  system  performance. 
This  comparison  is  often  conducted  using  radar  error  models 
to  characterize  radar  performance. 

Over  the  last  decade,  several  studies  sponsored  by  the 
FAA  have  employed  a  single  Gaussian  distribution  to  model 
the  azimuth  jitter,  error  in  the  measurement  of  the  aircraft’s 
azimuthal  position  relative  to  the  radar  beam,  of  a  monopulse 
secondary  surveillance  radar  (MS SR)  [3].  However,  as  part 
of  a  recent  study  in  Europe  the  Radio  Technical  Commission 
for  Aeronautics  (RTCA)  Requirements  Focus  Group  (RFG) 
developed  a  new  radar  error  model  for  MSSR  azimuth  jitter 
that  focused  more  on  low  probability  errors,  the  tails  of 
the  error  distribution.  They  determined  in  their  analysis  that 
a  single  Gaussian  distribution  was  too  narrow  and  did  not 
adequately  model  low  probability  MSSR  azimuth  jitter.  The 
RFG  instead  chose  to  employ  a  double  Gaussian  distribution 
with  wider  tails  to  model  MSSR  azimuth  jitter  [4]. 

In  this  paper  we  evaluate  the  suitability  of  the  two  radar 
error  models  for  use  in  the  establishment  of  separation  stan¬ 
dards  by  comparing  them  to  a  large  collection  of  radar  errors. 
The  radar  errors  are  calculated  using  a  technique  developed 
at  M3T  Lincoln  Laboratory  to  estimate  radar  azimuth  jitter 
from  large,  widely  distributed  sets  of  radar  data.  For  our 
analysis  we  show  results  from  over  two  million  azimuth  error 
samples  collected  from  several  ASR-9  Mode  S  MSSR  within 
the  national  airspace  system  (NAS).  We  then  best  fit  single 
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and  double  Gaussian  distributions  to  the  sampled  error  and 
compare  them  to  the  established  error  models.  Finally,  we  fit 
a  third  distribution  type,  a  Gaussian  Laplace  (or  Gaussian- 
exponential)  distribution  to  the  error  distribution,  and  compare 
it  to  the  other  models  through  various  metrics. 

II.  Background 

A.  Radar  Azimuth  Errors 

Radar  azimuth  errors  consist  of  a  combination  of  two 
different  errors:  azimuth  measurement  error  and  radar  azimuth 
bias.  Azimuth  measurement  error,  or  azimuth  jitter,  is  the 
error  in  the  measurement  of  the  aircraft’s  azimuthal  position 
relative  to  the  radar  beam.  Radar  azimuth  bias  is  a  systematic 
error  caused  by  a  misalignment  of  the  radar  with  true  north. 
In  the  existing  pre-ADS-B  single  sensor  radar  system,  radar 
azimuth  bias  has  minimal  effect  on  surveillance  performance 
with  respect  to  separation  services.  Single  sensor  radar  systems 
apply  a  sensor’s  azimuth  bias  equally  to  all  aircraft  and  the 
bias  is  in  effect  canceled  out  in  any  separation  measurements. 
Consequently,  when  using  single  sensor  radar  as  a  reference 
system,  the  system  and  its  corresponding  radar  error  models 
are  usually  assumed  to  have  no  azimuth  bias.  All  radar  errors 
and  error  models  mentioned  in  this  paper  refer  solely  to  the 
azimuth  measurement  error  and  do  not  account  for  any  radar 
azimuth  bias. 


B.  Single  Gaussian  Error  Model 

The  traditional  single  Gaussian  model  for  radar  azimuth 
jitter  is  a  normal  distribution  with  zero-mean.  Its  probability 
density  function  (PDF)  is  defined  as  follows: 


/(*)  = 


;  e  2^5 
y/2ncr2 


(1) 


The  standard  deviation,  cr,  for  this  MSSR  azimuth  error  model 
is  accepted  to  be  0.068°  (0.8  Azimuth  Change  Pulse)  [5], 


C.  Double  Gaussian  Error  Model 

The  RFG  determined  that  the  single  Gaussian  distribution 
underestimated  the  low  probability  MSSR  azimuth  errors. 
They  developed  a  new  error  model  using  a  double  Gaussian 
distribution,  the  weighted  sum  of  two  normal  distributions. 
The  model  consists  of  a  heavily  weighted  and  narrower  normal 
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distribution,  similar  to  the  single  Gaussian  error  model,  and 
a  lightly  weighted  and  wider  normal  distribution  intended  to 
characterize  larger  low  probability  azimuth  errors.  The  model 
is  mathematically  defined  as  follows: 


f(x)  =  (l-a) 
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where  the  accepted  value  of  the  weighting  factor  a  is  .05, 
and  the  standard  deviations  and  <r2  are  .054°  and  0.27° 
respectively  [4]. 


D.  Gaussian  Laplace  Distribution 

We  evaluated  a  third  distribution  type,  a  Gaussian  Laplace, 
as  a  potential  model  for  MS  SR  azimuth  error.  Similar  to 
the  double  Gaussian  error  model,  the  Gaussian  Laplace  is  a 
weighted  sum  of  two  distributions;  a  narrow  normal  distri¬ 
bution  to  model  typical  smaller  azimuth  errors  and  a  wider 
distribution,  in  this  case  a  Laplace  or  double  exponential 
distribution,  to  model  larger  low  probability  azimuth  errors. 
The  functional  form  of  the  Gaussian  Laplace  distribution  is 
defined  as: 

/<I)=<I~“)vi?e‘*+“se'151  <3) 

where  a  is  the  weighting  factor,  o  is  the  standard  deviation  of 
the  normal  distribution,  and  A  is  the  scaling  parameter  of  the 
Laplace  distribution. 


E.  Azimuth  Error  Estimation  Technique 

The  radar  error  estimation  technique  developed  at  MIT 
Lincoln  Laboratory  accurately  estimates  radar  azimuth  jitter 
using  raw  time-stamped  range/azimuth  radar  reports  from  air¬ 
craft  flying  through  the  NAS,  known  as  targets  of  opportunity 
(TOO),  providing  an  efficient  solution  for  analyzing  large 
quantities  of  radar  data.  The  method  filters  TOO  data  so  that  it 
contains  only  aircraft  flying  straight  and  level  at  a  near  constant 
velocity,  and  then  uses  this  a  priori  knowledge  of  the  aircraft 
behavior  to  accurately  estimate  the  true  aircraft  position  and 
radar  error. 

The  estimation  technique  involves  a  multi-step  process. 
First,  tracks  of  raw  secondary  reports  from  a  single  sensor 
recorded  in  radar  coordinates  (azimuth,  slant  range,  and  alti¬ 
tude)  are  projected  onto  a  stereographic  plane  and  stored  in 
Cartesian  coordinates  (x,  y)  relative  to  the  sensor.  Each  track 
is  then  passed  through  a  filter  that  calculates  the  smoothed 
heading  and  velocity  of  the  aircraft  and  extracts  periods  of 
straight  and  level  flight  at  a  near  constant  velocity.  The  straight 
and  level  tracks  are  then  fed  to  the  estimation  algorithm. 
The  algorithm  estimates  the  true  trajectory  of  the  aircraft  as 
the  line  that  best  fits  the  track  in  a  least-squares  sense.  The 
algorithm  estimates  the  true  position  of  the  aircraft  at  the  time 
of  each  radar  measurement  by  finding  the  set  of  points  (xit  y{) 
on  the  least-squares  line  that  minimize  the  summed  distance 
between  the  points  and  the  measurements  (a:,,  yi),  under  the 
constraint  of  the  aircraft  flying  at  a  constant  groundspeed  v. 


TABLE! 
ASR-9  Sensors 


Site  Location 

Site  ID 

Baltimore,  MD 

BWI 

Chicago,  EL 

ORD 

Boston,  MA 

BOS 

Los  Angeles,  CA 

LAX 

Manchester,  NH 

MHT 

New  York,  NY 

JFK 

Newark,  NJ 

EWR 

More  precisely,  we  are  in  search  of  the  following  set: 


{(si.J/lMmin^di: 

xi’Vt  i=1 
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where  dt  is  the  distance  between  the  points  (i4,  yi)  and 
(xi>  J/*),  and  fi+1and  ti  are  the  time  stamps  of  consecutive 
radar  reports.  The  estimated  true  positions  are  then  converted 
back  into  radar  coordinates  and  the  azimuth  jitter  is  calculated 
for  each  measurement. 

A  detailed  explanation  and  validation  of  the  estimation 
technique  is  found  in  [6]. 


III.  Analysis 

A.  Data  Collection 

Radar  data  was  supplied  by  the  84th  Radar  Evaluation 
Squadron  (RADES).  The  data  set  consisted  of  over  7  million 
radar  reinforced  beacon  reports  collected  from  8  different 
ASR-9  sensors  over  a  period  of  14  days  from  February  28th 
to  March  13th,  2009.  Table  1  lists  the  ASR-9  sensors  chosen 
for  the  analysis.  The  radar  reports  consisted  of  range,  azimuth 
and  pressure  altitude  measurements.  Prior  to  projection  onto  a 
stereographic  plane  the  pressure  altitude  measurements  needed 
to  be  corrected  using  local  weather  data.  Aircraft  altitude 
is  reported  as  Mode  C  pressure  altitude  from  sea  level  on 
an  average  day,  barometric  pressure  of  29.92  mm  Hg.  As 
local  barometric  pressure  deviates  from  this  standard.  Mode 
C  altitude  reports  become  inaccurate.  To  ensure  accurate 
position  projections  Mode  C  altitude  reports  were  corrected 
to  the  true  altitude  using  the  local  hourly  barometric  pressure 
and  temperature.  Each  altitude  corrected  radar  report  was 
then  projected  onto  a  stereographic  plane  tangential  to  its 
sensor  and  stored  in  a  database  in  both  radar  and  Cartesian 
coordinates. 


B.  Model  Fitting  Technique 

Three  different  distributions  were  fitted  to  the  estimated 
azimuth  errors  and  evaluated  for  accuracy.  Because  only  radar 
azimuth  jitter  was  estimated,  the  error  distribution  is  zero 
mean,  and  therefore  the  fitted  distributions  are  assumed  to  be 
zero  mean  as  well.  The  functional  form  of  the  three  different 
distributions  and  the  unknown  parameters  that  need  to  be 
estimated,  a,  <rx,  and  A,  are  listed  in  Eq.  (1-3). 

A  sample  PDF  of  the  estimated  azimuth  error  was  created 
by  counting  the  frequency  of  occurrence  of  azimuth  errors 


Fig.  1.  Estimated  Azimuth  Error  vs  Established  Error  Models 


Fig.  2.  Estimated  Azimuth  Error  vs  Best  Fit  Error  Models 


in  bins  of  size  0.01°  ranging  from  -0.5°  to  0.5°  .  Then  the 
distribution  was  normalized  by  dividing  the  count  in  each  bin 
by  the  total  number  of  samples.  The  best  fit  parameters  for 
each  distribution  type  were  determined  from  a  least-squares 
minimization  of  the  sum  of  the  differences  between  the  model 
PDF  value  and  estimated  error  PDF  value  at  each  bin  location. 

C.  Model  Evaluation  Metrics 

Models  were  evaluated  for  overall  goodness  of  fit  using 
the  coefficient  of  determination  or  R2  value.  Additionally 
the  estimated  error  PDF  was  broken  down  into  a  bands  (i.e. 
|0  to  crj,  |cr  to  2cr|,  etc.)  and  each  model  was  evaluated  for 
quality  of  fit  in  each  band  using  the  following  metrics: 


•  Probability  difference 

|Pr(model  PDF)  -  Pr(estimated  PDF)| 

•  Percent  difference 

|  Pr(modelPDF) —Pr(esti mated  PDF)  |  a  rto/ 

|Pr(model  PDF)+Pr(estimted  PDF)|/2  '  luu/0 

IV.  Results 

Figure  1  illustrates  the  comparison  between  the  estimated 
error  PDF  and  the  two  established  radar  error  models.  Figure 
2  shows  the  comparison  between  the  estimated  error  PDF  and 
the  best  fit  single  Gaussian,  double  Gaussian,  and  Gaussian 
Laplace  distributions.  The  locations  of  a  values  beyond  |3<t| 
were  left  out  for  the  sake  of  clarity.  The  results  in  each 
figure  are  plotted  on  both  linear  and  semi-log  scales,  the 
upper  and  lower  plots  respectively.  The  parameters  for  each 
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distribution  are  shown  in  the  legends  of  the  upper  plots 
while  the  R2  goodness-of-fit  statistics  are  shown  in  the  lower 
plots’  legends.  The  detailed  model  evaluation  metrics  for  each 
distribution  are  shown  in  Figure  3,  where  the  measured  and 
model  probabilities  refer  to  the  probability  from  the  specified 
PDF  of  an  error  occurring  in  each  a  band. 

The  results  confirm  the  conclusion  of  the  RFG  study  [4]  that 
the  single  Gaussian  error  model  does  not  capture  the  larger  low 
probability  MSSR  azimuth  errors.  For  work  where  these  low 
probability  MSSR  errors  are  of  concern  a  distribution  with 
wider  tails  (e.g.  double  Gaussian  or  Gaussian  Laplace  error 
model)  would  be  more  appropriate.  This  analysis  found  the 
best  fit  Gaussian  Laplace  error  model  to  be  more  accurate  than 
the  best  fit  double  Gaussian  error  model,  especially  at  very 
low  probabilities,  greater  than  4cr  from  the  distribution  mean. 
However,  the  relative  sample  sizes  at  these  low  probabilities 
are  very  small  and  the  difference  between  the  performance  of 
the  two  models  is  on  the  same  order  as  the  estimation  noise. 

In  applications  where  only  the  nominal  performance  of  the 
radar  is  of  concern,  azimuth  errors  between  —2 a  and  2cr  of 
the  mean  of  the  error  distribution,  a  single  Gaussian  error 
model  might  be  adequate.  However,  the  current  established 
single  Gaussian  model  uses  a  statistical  standard  deviation  of 
the  azimuth  error,  0.068°,  as  the  standard  deviation  of  the 
single  Gaussian  distribution  which  in  this  analysis  was  shown 
to  overestimate  the  nominal  azimuth  error.  A  best  fit  single 
Gaussian  similar  to  the  one  presented  in  this  paper  would  be 
a  more  conservative  model  and  might  be  more  appropriate  for 
reference  system  analysis. 


V.  Conclusion 

A  large  scale  analysis  was  performed  on  ASR-9  data  to 
evaluate  different  azimuth  error  models.  The  results  confirmed 
that  a  single  Gaussian  error  model  does  not  capture  larger  low 
probability  MSSR  azimuth  errors  and  that  a  double  Gaussian 
or  Gaussian  Laplace  distribution  would  be  more  appropriate 
for  modeling  those  errors.  However,  when  modeling  nominal 
radar  performance  it  was  found  that  a  single  Gaussian  dis¬ 
tribution  is  adequate  and  a  best  fit  Gaussian  might  be  more 
appropriate  for  reference  system  analysis  as  it  would  be  a  more 
conservative  error  model. 
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